NCPTL: A Network Correctness and Performance Testing Language

This paper introduces a new, domain-specific specification language called CONCEPTUAL. CONCEPTUAL enables the expression of sophisticated communication benchmarks and network validation tests in comparatively few lines of code. Besides helping programmers save time writing and debugging code, CONCEPTUAL addresses the important— but largely unrecognized—problem of benchmark opacity. Benchmark opacity refers to the current impracticality of presenting performance measurements in a manner that promotes reproducibility and independent evaluation of the results. For example, stating that a performance graph was produced by a “bandwidth” test says nothing about whether that test measures the data rate during a round-trip transmission or the average data rate over a number of back-toback unidirectional messages; whether the benchmark preregisters buffers, sends warm-up messages, and/or pre-posts asynchronous receives before starting the clock; how many runs were performed and whether these were aggregated by taking the mean, median, or maximum; or, even whether a data unit such as “MB/s” indicates 106 or 220 bytes per second. Because CONCEPTUAL programs are terse, a benchmark’s complete source code can be listed alongside performance results, making explicit all of the design decisions that went into the benchmark program. Because CONCEPTUAL’s grammar is English-like, CONCEPTUAL programs can easily be understood by non-experts. And because CONCEPTUAL is a high-level language, it can target a variety of messaging layers and networks, enabling fair and accurate performance comparisons.